An increasing number of video cameras are equipped with CMOS sensors instead of CCD sensors to reduce cost. Most CMOS sensors use rolling shutters that expose scan lines of a frame sequentially, in either top-to-bottom or bottom-to-top order. If the camera or objects in the scene being captured using a CMOS sensor move, distortions due to differences in exposure times, typically referred to as rolling shutter distortions, may be observed. Rolling shutter distortions may include, for example, wobble distortion, skew distortion, and vertical scaling distortion. While rolling shutter distortions caused by local object motion can be very complicated, in general, and especially for hand-held cameras, distortions due to global camera motion are dominant. Accordingly, prior art techniques for reducing rolling shutter distortion typically only consider distortions caused by global camera motion. Examples of prior art techniques are described in the following: C. Geyer, et al., “Geometric Models of Rolling-Shutter Cameras,” IEEE Workshop on Omnidirectional Vision, 2005; W.-H. Cho and K.-S. Hong, “Affine Motion Based CMOS Distortion Analysis and CMOS Digital Image Stabilization,” IEEE Transactions on Consumer Electronics, vol. 53, issue 3, pp. 833-841, August 2007; C.-K. Liang, et al., “Analysis and Compensation of Rolling Shutter Effect”, IEEE Transactions on Image Processing, pp. 1323-1330, August 2008; J.-B. Chun, et al., “Suppressing Rolling-Shutter Distortion of CMOS Image Sensors by Motion Vector Detection,” IEEE Transactions on Consumer Electronics, pp. 1479-1487, November 2008; and D. Bradley, et al., “Synchronization and Rolling Shutter Compensation for Consumer Video Camera Arrays,” International Workshop on Projector-Camera Systems, 2009.
Global camera motion may be a combination of translational and rotational motion. Many techniques for reducing rolling shutter distortion assume that the global motion is constant between adjacent frames since the intervening time interval is normally small. For hand-held cameras, horizontal camera panning, vertical camera panning, and camera shaking are typical global translational motions. As analyzed in the above cited references, horizontal panning may cause skew distortion, i.e., the frames are bent to one side. Vertical panning results in vertical scaling distortion, i.e., the frames are stretched out or shrunk vertically. Rotational motion may also cause non-linear rolling shutter distortion.
Prior art techniques for reduction rolling shutter distortion, e.g., the above cited references, have considered the compensation of skew and vertical scaling artifacts based on the assumption that the translational motion intended by the user (e.g., panning, zooming, walking toward a subject) can be accurately estimated. But even if the camera is held still and the intentional translational motion is zero, camera shake still results in a type of rolling shutter distortion known as wobble distortion, a combination of oscillations in skew and vertical scaling. More specifically, horizontal shake may cause sudden and alternating skews to the left and right and vertical shake may cause alternating stretching and shrinking in the vertical direction.